Tests: adapt HTML escaping to pygments v2.21.0 change-in-behaviour - #14611

@jdillard

Just FYI, I'm pretty sure the visible output change came from pygments#3185 (look at the note at the bottom of the description).

Thanks - I've updated the PR description to link to that instead.

@picnixz a new CI test failure just dropped (you can see how it manifests in #14241 for example). It would be great to get this one merged, when you get a chance, before it starts to complicate PRs

@picnixz a small nudge/reminder in case this can be merged

Independent verification, in case it helps the review: on Windows (Python 3.13.13, Pygments 2.21.0), the currently-red set — test_viewcode, test_viewcode_linenos, test_default_highlight, test_html_code_role, and both test_additional_targets_* intl tests — fails on master-equivalent code and passes on this branch (commit bb31616). With Pygments 2.20.0 the same six pass on both, consistent with the green Oldest supported job, so the version-conditional expectations hold on both sides of the change.

For context: every open PR's test matrix is currently red on exactly this set (the runs on #14654, #14662, and #14666 all show the same six failures), so merging this would restore review signal across the board. cc @picnixz

Gentle follow-up: this has been approved for a month now, and the Pygments 2.21 failures it fixes are still red across much of the open PR queue, including the viewcode PRs and our #14666. The Windows verification across Pygments 2.20 and 2.21 from early September still holds on current master. I also see #14674 covers part of the same ground. No preference from me on which route lands, but getting one of them in would unblock a lot of CI. Is there anything holding it up that an outside contributor could help with?

Ah sorry, I'm just overwhelmed by my daily job and travels that I don't have enough time for OSS. One comment though: can you refactor the pygments version split and add some PYGMENTS_VERSION_INFO = tuple(...) in some util module that we could import from everywhere in the tests instead of having to recompute the version info tuple in every test? for now I'm going to merge this but please make a follow-up if you can find the time.

Thanks @picnixz - yep, I'll take a look into refactoring that at some point (I can't promise when, I'm finding it hard to find time to focus here at the moment too). I'll also compare to how any other library/version checks are performed in the test suite.

I see that docutils and latex are still failing. Is there a PR I can merge for that?

@picnixz

I see that docutils and latex are still failing. Is there a PR I can merge for that?

Pull request #14673 from @adamtheturtle resolves most of the docutils HEAD issues.

⚠️ However...

When I run that locally today (using the docutils mirror on GitHub -- because repo.or.cz is geoblocked from the United Kingdom) the remaining failures are:

1. HTML output tests where RFC URLs, instead of emitting .html suffixes, emit empty-directory (trailing slash) URLs:

2. A failure in test_cfunction where the following diff occurs:

E         Full diff:
E           [
E               '',
E         -     '.. py:function:: asctime([tuple]) -> string',
E         +     '.. py:function:: asctime([time_tuple]) -> string',
E         ?                                 +++++
E               '   :module: time',
E               '',
E               "   Convert a time tuple to a string, e.g. 'Sat Jun 06 16:26:11 1998'.",
E               '   When the time tuple is not present, current time as returned by '
E               'localtime()',
E               '   is used.',
E               '',
E           ]

tests/test_ext_autodoc/test_ext_autodoc_autofunction.py:126: AssertionError

@jayaddison FWIW I don't have time right now to contribute further to that, but I take no offence at someone taking my work and getting it to shippable!

Ok, thanks @adamtheturtle. Please could you rebase and/or merge-from-master into the branch so that the latest CI status is more clearly visible?

Thanks!

Self-quoting:

2. A failure in [`test_cfunction`](https://github.com/sphinx-doc/sphinx/blob/b04a2101295ac3fb725b16111eda0284b6da4cca/tests/test_ext_autodoc/test_ext_autodoc_autofunction.py#L124-L135) where the following diff occurs:

Note: that failure has been reported resolved already in #14683 (thanks @mitya); it's due to the fact that python3.14 in Debian testing (that I ran the tests with) includes python/cpython#155790

I'll try to take a look into resolving that soon too (although it may be tricky to use a Python system version as a range boundary, since no formal tagged release of cPython yet includes that change..)

Edit: thread #14683 is a pull request, not an issue report thread as I initially thought. It includes a fix, and the fix looks good to me.

Hej, @picnixz the Docutils HEAD failures are part of currently the Docutils 1.0 breaking changes being merged on dev one by one, so even with current fixes, the failures will continue, until the possible interceptions have landed and/or are resolved.
I have been trying to open issues one by one as they are merged, but for many we have to wait for the upstream merge to see the final implementation to test against.
In #14680 I made this small table of what I could find in the Docutils changes that affect Sphinx (In line with the AI Policy: I used the help of AI to aggregate all the open already issues and PRs and summarize upstream changes) (Also, the last one is a catch all, many of the changes dont affect sphinx itself (or shouldn't according to what I can see) but could add-ons, as there are some node changes etc.)
I added the PR by @jayaddison to the table (#14696 provides a partial fix to #14680 and doesnt fix #14659, adjusting tests to pass for now.) (@jayaddison for the RFC changes I suggest to decouple partially from Docutils, as the current construction seems to be an artifact from development to make the construction version agnostic across versions as the new notation is the correct one going forward and should replace the existing one. For the table changes we have to be a bit more careful with cross version behavior with fractional/proportional widths (which I think will be bigger piece of writer work for Sphinx going towards Docutils 2.0) and indented doctests, also to not make current edge behavior programmatically dependent. If you want you can look at my PRs #14681 and #14689 how I suggest to fix them.):

As this will continue, the question will be whether for now Docutils HEAD tests should be paused and/or if we want to open a tracking issue, as there is a bit of work open that has been announced in the Docutils Release Notes that will land in the coming days to weeks.

Disclaimer, I also wrote the PRs #14681, #14689.

Purpose

This pull request should allow the Sphinx Python unit test suite to pass; it began failing recently following the release of Pygments v2.21.0 that included adjustments to HTML quote-escaping logic: pygments/pygments#3185

This changeset adapts the relevant unit test expectations according to the runtime version of Pygments detected.

References

• Resolves ci: unit test failures due to HTML escaping change #14610

AI Disclosure

I did not use any AI/LLM tooling for this pull request; neither to develop the changes locally, nor to create the pull request itself.

---

Edit: correction: a different Pygments PR seems to have been the cause of this; the hyperlink has been updated
Edit: rephrase for conciseness/clarity